I think a lot of advance is taken from shear scale and would wonder what the numbers are of the items you mention in your blog post. We are in the process of doing what you have already done and the upfront investment is quite high with no real quantifiable model to calculate the outcome.... we are doing it because we think/assume this is the best way forward. OK we have very clever/experienced people here so there is some justification... but wonder how long it took Google to achieve this "testing nirvana".
Thank you so much for sharing this with us Anthony,
While reading it I did have a question that came to mind. You mention "simulating tens of thousands of users after just minutes of prep work". I'm very curious as to what technology you're using to simulate the users? Is it something like a selenium webdriver running a real browser or maybe a headless browser like phantom.js or htmlunit? Or are you using an HTTP based simulation like JMeter or Gatling?
I work for a company that provides JMeter as a service so we ourselves have dealt quite a bit with large scale distributed testing and I'd love to hear what technology Google is using in-house :)
We have several internal load testing frameworks to choose from, covering various languages and use cases. In my case, I was testing Google Cloud Storage, so I needed one that was geared toward testing a RESTful service. The selected framework is based on FunkLoad (http://funkload.nuxeo.org/). We have other frameworks that are based on JMeter as well. All of these frameworks depend heavily on the job management tools mentioned above in order to start, monitor, and stop load generating jobs on many machines.
Hi, Anthony Thank you for sharing your experience about testing. Like you said in the article, you are using actually two sorts of test framework to validate restful service such as funkload, jmeter, etc. I guess that your testing is to aim to validate load based performacne/stress on restful service. On the other hand, I'm wondering how you are testing restfull API with what kind of test framework.
Its always useful for small companies, 1 person even, to invest in tooling to make coding / configuration management / testing / deploying / easier and less error prone. But its not something you necessarily do day 1 ( some things you do straight from the start), its something you do when there's enough friction that you could see a tool would be useful, but before there's so much friction that you are hampered by "manual" process, or that you have no facility to do something ( like a particular kind of testing).
Thanks Anthony, A nice and informative post. If we look the development scenario from a startup stands, it will be little costly but once they have products development in line they can move to this implementation. Still customers prefer to deliver more in less time so spending time in non-functional testing unless asked by customer is not a good choice.
But at the other side one implementation can be done as automation of code testing during off hours. Like team can submit the code at the end of the day and all of those codes can be executed with automated deployment and testing strategy with the tools like Jenkins. On completion of execution testing framework should send email for test pass/fail status.
I have a basic question, how test cases creation is done in google. Are you engaging you talented resources in manual mundane task of test case creation.
A very informative post. I have a couple of question. Where does a feature get tested in Google? To elaborate more, I would ask, Is the feature get tested in feature branch and then merged to master? What is the exact process?
Hi Anthony,
ReplyDeleteI think a lot of advance is taken from shear scale and would wonder what the numbers are of the items you mention in your blog post. We are in the process of doing what you have already done and the upfront investment is quite high with no real quantifiable model to calculate the outcome.... we are doing it because we think/assume this is the best way forward. OK we have very clever/experienced people here so there is some justification... but wonder how long it took Google to achieve this "testing nirvana".
Cheers
Thank you so much for sharing this with us Anthony,
ReplyDeleteWhile reading it I did have a question that came to mind. You mention "simulating tens of thousands of users after just minutes of prep work". I'm very curious as to what technology you're using to simulate the users? Is it something like a selenium webdriver running a real browser or maybe a headless browser like phantom.js or htmlunit? Or are you using an HTTP based simulation like JMeter or Gatling?
I work for a company that provides JMeter as a service so we ourselves have dealt quite a bit with large scale distributed testing and I'd love to hear what technology Google is using in-house :)
Thanks,
Ophir from BlazeMeter
Hi Ophir,
DeleteWe have several internal load testing frameworks to choose from, covering various languages and use cases. In my case, I was testing Google Cloud Storage, so I needed one that was geared toward testing a RESTful service. The selected framework is based on FunkLoad (http://funkload.nuxeo.org/). We have other frameworks that are based on JMeter as well. All of these frameworks depend heavily on the job management tools mentioned above in order to start, monitor, and stop load generating jobs on many machines.
-Anthony
Hi, Anthony
DeleteThank you for sharing your experience about testing. Like you said in the article, you are using actually two sorts of test framework to validate restful service such as funkload, jmeter, etc. I guess that your testing is to aim to validate load based performacne/stress on restful service. On the other hand, I'm wondering how you are testing restfull API with what kind of test framework.
Regards,
Jeong Jin-ho
Its always useful for small companies, 1 person even, to invest in tooling to make coding / configuration management / testing / deploying / easier and less error prone. But its not something you necessarily do day 1 ( some things you do straight from the start), its something you do when there's enough friction that you could see a tool would be useful, but before there's so much friction that you are hampered by "manual" process, or that you have no facility to do something ( like a particular kind of testing).
ReplyDeleteThanks Anthony,
ReplyDeleteA nice and informative post. If we look the development scenario from a startup stands, it will be little costly but once they have products development in line they can move to this implementation. Still customers prefer to deliver more in less time so spending time in non-functional testing unless asked by customer is not a good choice.
But at the other side one implementation can be done as automation of code testing during off hours. Like team can submit the code at the end of the day and all of those codes can be executed with automated deployment and testing strategy with the tools like Jenkins. On completion of execution testing framework should send email for test pass/fail status.
I have a basic question, how test cases creation is done in google. Are you engaging you talented resources in manual mundane task of test case creation.
ReplyDeleteA very informative post. I have a couple of question. Where does a feature get tested in Google? To elaborate more, I would ask, Is the feature get tested in feature branch and then merged to master? What is the exact process?
ReplyDelete