I will add this:

I have a different ‘pre_get_posts’ filter to only show posts where a custom meta field is set to either 4 or 5. Again, this works for the first page of posts. When the infinite scroll kicks it, it just starts loading posts using the default query. (all posts, reverse chronological order).

In short, Infinite Scroll always uses the default query for the archive and does not apply the filter. My posts per page is 18, and the second group of posts is the second group of 19 through 36 as if there was no filter applied.