So my team and I have struggled for some time now with using the Email Security Service message log search/advanced search filter. Different search parameters that you really think should work, simply didn't. I was never overly impressed with the search functionality, but we just tried various conditions to get the results that we were looking for, or so we thought. What I uncovered just recently, has made me lose complete confidence in this solution. I'm going to walk through the steps that I undertook to discover this issue and will attempt to describe it, but it's really best seen/performed in person, so I highly encourage everyone reading this to attempt some variation of the steps below yourselves.
A couple weeks ago, my team and I performed a Phishing campaign on the rest of our staff utilizing a third-party service. We had provided to them a list of all of the staff accounts who were to be phished (roughly 600). With all the necessary white listing in place, we kicked off the campaign. The emails were all being sent at once, as opposed to trickling in. As the emails started flowing in, I wanted to monitor the progress to make sure there weren't any issues (rate-limiting, etc.) so I pulled up the Email Security portal and clicked over to the message log page. After a few minutes of watching them come through, they eventually stopped, indicating that they all should have been sent by that point. It was a whole lot of emails, so I wanted to take a closer look at just those and make sure they all made their way through. My first log search resulted in roughly 420 results, indicated in the top right of the results window. That was definitely way off from the expected count of 600, so I changed the page view to 200 results at a time to check timestamps and such. I made my way through to the third results page (of 3) and nothing had really jumped out at me. At this point, I decided to simply export the search results and perform a lookup comparison of the accounts from my original spreadsheet, so that I could see what emails might not have made their way through. I should note that the export function only exports the current page of results, not the entire search result. Therefore, I had to perform 3 exports for the 3 pages of results. I compiled the two lists for comparison and noticed that my own account wasn't shown even though I knew that I had personally received the phishing email as well. I perform a targeted search on myself and it showed up in the results. Great! So why didn't it show up in the initial search? I go back to my original search filter (just sending address) and make my way through all three pages a bit more closely. WTH? I'm not in there. Just to confirm I didn't miss it, I clicked on the previous results page button. Once loaded, my total result count in the top right of the page now indicated 380. Wait a second....simply navigating between the results pages is actually changing my result count??? I don't see my account listed at all on the second page, so I click back to the first.....there I am! I knew something was definitely up. At this point I'm really questioning what I'm seeing, so I figured I would log out, close out my browser session, and try going back in to re-perform the search. The second time through, I utilize the text search feature of my browser just to make sure that I wasn't somehow overlooking my account name. First page, I'm not there. Second page, nothing. Third page, the same. I now click the previous page button and my total result count changes once again. I'm still not listed, so I click back to the first page of results. There I am again!
I proceed to randomly select a handful of other user accounts that were not listed in the export files and individually target them in a search. They all come up. Thinking this may be some funky browser caching issue (long shot here), I try in total, Chrome, Firefox, and Edge, all with the same exact result. This is CLEARLY a back-end issue with the cloud service.
I opened a support ticket (#02857608) and established a remote session with a technician who indicates that he had seen similar oddities in the past, but hadn't ever been able to really put his finger on what was going on. He was astounded by what I showed him and agreed that it was a significant issue with the service. He indicated that he was going to escalate the issue and we left it at that. That ticket was closed shortly thereafter with zero follow up.
This is a huge issue! My team and I frequently use the message log filters to help quickly identify true phishing attempts, malicious emails, and for many other functions where accuracy and dependability are paramount to what we, as IT, do every day. This issue has broken that fundamental trust which is needed to use it as a critical tool. If we can't consistently trust that the results are accurate every time, it's of little value. I truly hope that this is investigated thoroughly and quickly resolved, because as it currently stands, the message log search function is broken.