Best Buy pause before checking if in-stock (try 2)

[rodalpho]

Since the other one was closed, I assume you didn't see my response. I'm fine if you label it WONTFIX, but please be aware it is not a duplicate. I'm aware of the in stock wait time, and have it set all the way up to 15 minutes now because of false positive alerts on Best Buy. If those false positives were addressed that would no longer be necessary.

Original request:

On the awful Best Buy website, when you first load the page, it will often show yellow "add to cart" buttons, then a couple seconds later a secondary process refreshes and they properly show "sold out" grey buttons. Is it possible to implement a 4 second pause before checking if product shows in stock?

[geman220]

I don't see this behavior happening at all. I did get alerts for Best Buy today but the screenshots show Add to Cart and when I went to the page it also showed Add to Cart (consistent not for a brief second). My last Best Buy notification was just about an hour ago (10:37 am specifically).

[rodalpho]

I see the behavior I described every time they go in and out of stock. The botrtx guy just added a pause for best buy for this very reason, to reduce the false positives on that site. My last bestbuy notificaton was 2:03PM ET, if that helps.

[kjack9]

I also see this occasionally. Best Buy seems to have several "out of stock" behaviors:

• The button just says "Out of Stock" (currently handled)
• The button says "Add to cart", but a few seconds after the page loads it changes to "Out of Stock" (sometimes produces false-positives)
• The button says "Add to cart", but during the checkout process they determine that they have no inventory (not currently handled)

[rodalpho]

Right, I see no way to handle your 3rd case. That'll be a false positive every time. But hopefully we can eliminate the middle one.

[datbird]

Here's a GIF of the best buy page loading and eventually switching to "out of stock" as everyone above is referring to:

[geman220]

Here's a GIF of the best buy page loading and eventually switching to "out of stock" as everyone above is referring to:

That's very odd, that doesn't happen to me at all (just tried several F5s and new tabs/browsers.) Just out of curiosity what is your ping to Best Buy? I want to identify if this is a Best Buy site issue (as in we can safely predict the time it takes to do this in/out stock change) or if this is a network request issue (which means some sort of .env variable is probably best)

[datbird]

To be fair after doing this... alot I see it occasionally behaves differently off and on. I'll go literally days with out seeing it do that, then I'll get a single browser session where it does that, then almost all my browser sessions do it... etc Its not 100% consistent in my experience.

[rodalpho]

I have an 8ms ping to bestbuy.com (akamai CDN 23.5.132.113). So not a network issue.

[geman220]

I have an 8ms ping to bestbuy.com (akamai CDN 23.5.132.113). So not a network issue.

Hm, yea I have 30ms but I cannot replicate this at all. Does yours take as long as @datbird to change? Looks like ~10 seconds. The reason I thought it could be some sort of request issue is due to the .gif showing new elements loading for quite a while.

[kjack9]

This was especially happening this morning when Best Buy briefly had some Gigabyte stock. I'd guess it has to do with a caching issue on Best Buy's side.

[rodalpho]

No, it currently takes maybe 2-3 seconds to switch to sold out. That's why I suggested a 4 second pause.

[datbird]

I did have alot going on my PC at the time. Here is another with some more details and a not bogged down PC:

[neotron]

As a note, this isn't new with Best Buy. I've seen the exact behavior with the site when trying to buy a Nintendo Switch some months ago. Definitely does seem to happen when they have had stock, but it's out.

Most likely if you turn on network monitoring, there's a request that takes a while to finish or something like it.

[geman220]

I did have alot going on my PC at the time. Here is another with some more details and a not bogged down PC:

Hm, that makes it a bit more difficult also because it seems to be related to PC. Because your original .gif was ~10s and this new .gif with less CPU load seems to be ~3s. Which means it's not necessarily a flat value of "check again at this time" because it will depend on CPU load.

[rodalpho]

True enough, but adding a flat 4s pause would help a great deal. If you get ambitious, perhaps make it user-configurable.

[geman220]

True enough, but adding a flat 4s pause would help a great deal. If you get ambitious, perhaps make it user-configurable.

Yea no I agree, but it will be at least a fair amount of logic (as in not just 2 lines of code ez pz done), so at that point we might as well make it a .env variable instead of hardcoding a time. Just trying to brainstorm the best possible solution so we aren't constantly tweaking.

[thegutuproject]

Im not as well versed into the code base as I'd like to be, but could this be an issue with ad blockers/any sort of request interceptors messing up the refresh? I don't have this issue, at all, ever, but I also disabled my adblocker for the duration of this "witch hunt" for a gpu on vendor sites.

[datbird]

Thats a good point @thegutuproject . I use uBlock orgin, I am no longer seeing the issue so I cant test with it on/off to compare but that is a very good idea of something to nail down and ensure that's not what is causing the problem

[rodalpho]

What adblocker? This script uses a separate embedded copy of chromium.

Also to be clear, it happens without adblockers too-- I have a separate browser profile running with no adblocker just to be sure I can check out quickly when that blessed day comes, and the problem still occurs.

[geman220]

What adblocker? This script uses a separate embedded copy of chromium.

Also to be clear, it happens without adblockers too-- I have a separate browser profile running with no adblocker just to be sure I can check out quickly when that blessed day comes, and the problem still occurs.

I think the confusion is when it does open the browser that it's in "normal" chrome so it would seem like maybe adblocker is doing something. But you are right that the actual detection process is done with chromium and if this 'Buy Now' 'Sold Out' bug on their website exists normally, chromium is going to false-positive on it.

[geman220]

Just to add some more details. I just got the first Best Buy alert I've had since 6hrs ago. I went to https://www.bestbuy.com/site/gigabyte-geforce-rtx-3080-10g-gddr6x-pci-express-4-0-graphics-card-black/6430620.p?skuId=6430620 which is the detected site, but the Add to cart isn't changing to Sold out like the .gifs you have provided. I am not saying that in an argumentative way, just trying to log what's been happening. For some reason, I am not experiencing that weird Add to Cart switching to Sold Out.

Screenshot of successful product page

One thing I did notice, the specific site that is actually being scanned is https://www.bestbuy.com/site/gigabyte-geforce-rtx-3080-10g-gddr6x-pci-express-4-0-graphics-card-black/6430620.p?acampID=0&cmp=RMX&loc=Hatch&ref=198&skuId=6430620&intl=nosplash for example. It's not searching the entire 3080 list of products. Do you guys experience the same Add to Cart switching to Sold Out on the above product page?

Edit: Also is it only this gigabyte card? I don't seem to have any alerts for anything other than this card, isn't this the one you guys mentioned earlier?

[rodalpho]

Right, it didn't happen just now.

[geman220]

I'm also wondering if #294 might actually fix this. It will drop the network load by about 50% and could theoretically make the page load much faster and eliminate the issue. Might wait to see if it's merged and what the results are.

[zerocarbthirty]

Isn't that just a javascript thing that changes the stock from in stock to out of stock? what happens if you pull the page with js disabled?

[xninjax]

Isn't that just a javascript thing that changes the stock from in stock to out of stock? what happens if you pull the page with js disabled?

I came across the following post on handling a very similar problem (the need to wait for js to complete modifying the dom/html) https://stackoverflow.com/a/61304202/7240138

I think this may solve certain issues experienced with dynamic page content. Not certain it could solve the issues experienced here with bestbuy, but awaiting the page to settle (after load) might do the trick?

The stackoverflow post suggests using a simple html.length as indicator of 'page settled', but I wonder if a hash of the html would be more robust (or just plain overkill).

I might take this up and test - unless someone else wants to jump on it first 😃 👍

[jef]

but I wonder if a hash of the html would be more robust (or just plain overkill).

I might take this up and test - unless someone else wants to jump on it first 😃 👍

I think testing would be good here, because hashing an html and putting it in the stores object could report negative a lot. I'm sure the hash will change every time the page is reloaded (different texts, times, etc). Just a guess.

[xninjax]

Thanks @jef for the feedback.

I should clarify on my comment about the hashing though.

The suggestion on stackoverflow is to monitor the html for changes after page load for up to a defined timeout (say 5-10sec). If the html is still being modified during that timeout, continue checking in a loop (check every 500-1000ms). The method of determining "page changing" during this routine would be to get html.length each iteration and check against the previous html.length iteration, once the .length stops changing (for x number of consecutive iterations) we assume page has completed its modifications and move on.
That all said, I was thinking that instead of .length, "html.hash()" may be a more precise check between iterations. Although I too have concerns that a html hash may never truly settle? html.length may suffice.

But no intentions of storing a long term hash of the store pages - just as you pointed out, these are very dynamic over time.

Hope that clarifies my thought process here 😎

[jef]

Ahhh! This makes much more sense. Thanks for clarifying. Yeah! I totally think you should test that out. That sounds great.

[Eveets]

Getting the same issue on bestbuy.ca. This seems to happen to specific products (the same product will have a false positive repeatedly while others don't.) Maybe a delay setting in the .env for bestbuy only?