A couple of months ago I did a manual counting of how many times my homebrew versions were downloaded for each platform and posted it here. While I could do this every couple of weeks or every month...it's way to tedious and time consuming! And I am a developer so I created a programmers solution...what else! 
There is a tool called Selenium that can be used to automate testing of websites or web-applications. It uses a real browser to load the page and works with Chrome, Firefox, Edge and a couple of more exotic browsers. It's super useful to test something on all browsers and know it works just by running a script! That's the theory at least....I never used it for that (yet) since it's a powerful tool it also brings complexity. But for a few months the idea of using it for something kept going in my mind every now and then....
Then while in contact with the owners of BatteryCheck one of their requests was to provide download statistics of how much the homebrew version was downloaded for each platform I made a port for. So to make that easier I dived into Selenium and in a week or two I had a script that could:
- login to GBATemp
- "click" away the popups and cookie stuff
- go to each of the pages that have downloads
- go to my "all threads" page to see views.
Every page that is loaded with a Selenium controlled browser van be searched for various elements and texts. Very similar to whats possible with JQuery for those that know it. Searching for elements and text on the pages makes it possible to collect the views and download statistics and dump that info into a file. The idea is (was) to import that into a database and make a sort of dashboard thing to see how much views or downloads are made every day or week. All I did was create a second script that loads these "dumps" and show simplified statistics like this:
Here it counts the difference between the stats collected on June 22 and August 12. For me the difference between the two is the most important...as it shows the amount of interest there is in my project for each platform. And I was really surprised that it was the PS2 version that had the most downloads. Or that people are still downloading the earlier versions, hahahah. It's great to see and I want to say a big thank you to everyone for downloading or viewing the threads.
The stats are collected with Selenium as I said and it's controlled from a Python script that I have setup to run every three hours. The browser itself runs on my NAS inside a docker container so I could in theory even shut my PC down and the stats would still be collected. For now the script runs from my main PC though that's never turned off for various reasons......and I still had to move it into it's own docker container and run fully on the NAS. It completely possible to do this and it's planned but at the moment I am glad I had not done it yet....
Since a few days I noticed that the stats were "stuck" on Thursday at 18:00 (my local time) and that it did not resume later on it's own. I have had it a few times before that the remote browser got stuck on something and the script crashed. That requires a restart of the docker container that hosts the browser and all is good after that. Just to be sure I added a couple of extra delays to give the browser some time to load the pages.
This morning the stats were still stuck and I did some investigating on what might be causing it....then I noticed that there is a "cloudflare ddos protection page" loaded instead of GBATemp like normal. It seems that that check detects that it's a selenium browser and it denies access to the website! Since I did not get a direct response from an admin about collecting my stats in this thread I was assuming it was "ok" to get the stats this way. Ofcourse I understand why a protection like this is there....and I am not complaining about that. ok. just a tiny little bit maybe.
Hopefully I find a workaround and make my script work again.
There is a tool called Selenium that can be used to automate testing of websites or web-applications. It uses a real browser to load the page and works with Chrome, Firefox, Edge and a couple of more exotic browsers. It's super useful to test something on all browsers and know it works just by running a script! That's the theory at least....I never used it for that (yet) since it's a powerful tool it also brings complexity. But for a few months the idea of using it for something kept going in my mind every now and then....
Then while in contact with the owners of BatteryCheck one of their requests was to provide download statistics of how much the homebrew version was downloaded for each platform I made a port for. So to make that easier I dived into Selenium and in a week or two I had a script that could:
- login to GBATemp
- "click" away the popups and cookie stuff
- go to each of the pages that have downloads
- go to my "all threads" page to see views.
Every page that is loaded with a Selenium controlled browser van be searched for various elements and texts. Very similar to whats possible with JQuery for those that know it. Searching for elements and text on the pages makes it possible to collect the views and download statistics and dump that info into a file. The idea is (was) to import that into a database and make a sort of dashboard thing to see how much views or downloads are made every day or week. All I did was create a second script that loads these "dumps" and show simplified statistics like this:
Code:
GBATemp stats
TimeA: 2021-08-12 18:00:56
TimeB: 2021-06-22 18:02:47
Threads:
BatteryCheck - WII 16314 (2029)
BatteryCheck - GameCube 384 (103)
Batterycheck - 3DS 10031 (4112)
BatteryCheck - PS2 3094 (862)
BatteryCheck - WiiU 5081 (3659)
Batterycheck - PSP 3190 (53)
--------------------------------------------------
38094 (10818)
Files:
preview-batcheck-wii-v0.1.dol.zip 116 (2)
batterycheck-gc-preview-2.dol.zip 127 (7)
batterycheck-wii-preview-2.dol.zip 133 (7)
batterycheck-wii-preview-v0.3.zip 119 (10)
batterycheck-gc-preview-v0.3.zip 107 (4)
batterycheck-wii-v0.3.4-ALPHA.7z 11 (5)
batterycheck-gc-v0.3.4-ALPHA.7z 10 (7)
batterycheck-3ds-preview-2.3dsx.zip 207 (11)
bc_setup.3dsx.zip 78 (1)
batterycheck-3ds-installer-v0.3.zip 90 (6)
batterycheck-3ds-v0.3.4-ALPHA.7z 8 (1)
batterycheck-ps2-v0.3.2-ALPHA.elf.zip 36 (17)
batterycheck-ps2-v0.3.3-ALPHA.zip 30 (14)
batterycheck-ps2-v0.3.4-ALPHA.7z 30 (10)
batterycheck-wiiu-v0.3.6-ALPHA.7z 34 (15)
-----------------------------------------------------------------
1136 (117)The stats are collected with Selenium as I said and it's controlled from a Python script that I have setup to run every three hours. The browser itself runs on my NAS inside a docker container so I could in theory even shut my PC down and the stats would still be collected. For now the script runs from my main PC though that's never turned off for various reasons...
...and I still had to move it into it's own docker container and run fully on the NAS. It completely possible to do this and it's planned but at the moment I am glad I had not done it yet....Since a few days I noticed that the stats were "stuck" on Thursday at 18:00 (my local time) and that it did not resume later on it's own. I have had it a few times before that the remote browser got stuck on something and the script crashed. That requires a restart of the docker container that hosts the browser and all is good after that. Just to be sure I added a couple of extra delays to give the browser some time to load the pages.
This morning the stats were still stuck and I did some investigating on what might be causing it....then I noticed that there is a "cloudflare ddos protection page" loaded instead of GBATemp like normal. It seems that that check detects that it's a selenium browser and it denies access to the website!
Since I did not get a direct response from an admin about collecting my stats in this thread I was assuming it was "ok" to get the stats this way. Ofcourse I understand why a protection like this is there....and I am not complaining about that. ok. just a tiny little bit maybe.Hopefully I find a workaround and make my script work again.
