A short question w.r.t. SWEs writing APIs and the fakes. If writing fakes is painful enough to incentivize SWEs to write good and simple APIs, wouldn't the same force incentivize the SWEs to update the APIs? API design is hard to thing to get it right - even more so on the first attempt. If the SWEs are now responsible for updating Fakes (which I assume to be less than pleasant), wouldn't they shy away from improving the API?
Thanks for the question, taeold. I agree that all else being equal, if you make something more expensive it'll happen less.
Usually API changes are actually additions. It's hard to change the black box behavior of a service in a loosely-coupled system because you have to convince all clients to migrate to expect the new reponse.
Off the top of my head, API definitions or black-box behavior can change for several reasons:
1) The API leaks implementation details and the implementation needs to change for some reason. 2) A new feature is added. 3) The domain changes. (typically rare) 4) There is an error in the implementation that leaks bad API responses.
API owners are motivated by management performance reviews. For 2-4, they will be given poor marks if they do not respond to defects and changing business demands, so they will implement the changes and update the Fake despite the additional cost.
Almost all bad APIs I've seen are examples of 1. This is a symptom of a bad design and probably made the Fake hard to write in the first place. In a system that hides no implementation details, the Fake is as complicated as the original system. I would hope that teams in this situation would come to the awareness of their fundamental problem and fix it since that would lower long-term costs for everyone.
A few years ago I worked with a team whose primary query API sent clients the entire storage record, which was a kind of state machine snapshot, consisting of dozens of poorly-defined, nested, high-cardinality fields. Clients generally only needed one or two pieces of data, but they had little guidance as what was public and unchanging and often picked an internal, implementation-specific field to use. The service owners didn't know what clients needed because they were internally focused. Imagine how hard that API would be to support with a Fake and how motivated that team would be to learn what the domain actually was and implement an API using that knowledge if a realistic Fake was mandatory.
Dave
p.s. In case you're curious, the team eventually did move to a better API. They came to realize how much their API constrained their implementation since any internal change was a potential hard-to-debug defect only caught in integration tests.
I've neglected to think about the many other factors at play in a significant API change; substantial rewrite of a fake may only play a minor role. But the motivation write a good API to make it easy to write fakes is at full force in both initial design and painful redesign of the API.
Hi Dave - Thank you for writing out this post to minimize product-wide problems. In particular I was interested in Fakes as a solution to identify if an API is easy to use. I don't know if this would be the right place to inquire implementation details of Fakes but thought I would ask:
1) Is Fakes a good way for controlling the underlying persisted data(test data/fixtures) which the actual service use to generate the responses?
2) And if so, Should Fakes also implement ways to insert and delete test data/fixtures?
> 1) Is Fakes a good way for controlling the underlying persisted data(test data/fixtures) which the actual service use to generate the responses?
I think the answer to your question is entirely dependent on your architecture. The only hard-and-fast definition of Fakes is that they are simplified implementations of the service useful for functional testing.
I work on a product now where the external API is basically a filesystem as web service, so a good Fake at that level would definitely not control an underlying persistent datastore but would most likely use an in-memory representation. I've worked on a product where there are multiple levels of bidirectional-interacting services, some of which touch the same underlying datastores or otherwise have side communication channels such that you can't write a Fake of service A without some way of propagating state changes (side-effects) to the Fake of service B.
It's probable that there are domains where such a complex, coupled architecture is inevitable. In those cases, the Fakes would have to have some communication channel, and it might be by talking to a lightweight common storage. However, I think it's likely that when confronted with the complication of writing and maintaining a web of interacting Fakes, teams will realize that there are better architectural alternatives (e.g. isolated microservices) in many cases. In either case, it's not the Fakes that make life difficult, they simply put the pain that is felt by the clients onto the service owners.
> 2) And if so, Should Fakes also implement ways to insert and delete test data/fixtures?
I think based on my answer to 1) the general answer is "no". For test fixtures using Fakes I expect the Fakes to provide seeded data for read-only services and for the test to generate the test data during the setup phase by making writes to read/write services' Fakes. I don't expect any of that data to reach a real database like Bigtable or MySQL because such heavy-weight dependencies are inappropriate for functional tests.
My approach to test data in end-to-end (Fake-less) tests is to generate it using the public APIs of the system, since they are least likely to change in ways that cause the data to violate the data stores' schema and most likely to generate realistic data (aside from thorny issues such as legacy/migrated data.)
Hi David I enjoyed your post, with fakes in your experience would you design them to support prod like response times delays or is production response delays a feature more suitable more a mock stub framework? I am asking this from a performance testing point of view. Thanks again for the post
Hi John, I've helped SWEs add statistical delay strategies into fakes and that's worked fine. At Google we have a very standardized RPC system such that the often easier choice is to insert transparent, data-driven proxies that can fuzz timing as needed. In the case above the bespoke delay code was for performance testing a third-party, non-Google interaction so there was no existing infrastructure for inserting performance delays. If you have a common RPC mechanism, using dynamic proxies is the more general, lower-maintenance solution.
Thanks for a fun post - I enjoyed reading it.
ReplyDeleteA short question w.r.t. SWEs writing APIs and the fakes. If writing fakes is painful enough to incentivize SWEs to write good and simple APIs, wouldn't the same force incentivize the SWEs to update the APIs? API design is hard to thing to get it right - even more so on the first attempt. If the SWEs are now responsible for updating Fakes (which I assume to be less than pleasant), wouldn't they shy away from improving the API?
Thanks for the question, taeold. I agree that all else being equal, if you make something more expensive it'll happen less.
DeleteUsually API changes are actually additions. It's hard to change the black box behavior of a service in a loosely-coupled system because you have to convince all clients to migrate to expect the new reponse.
Off the top of my head, API definitions or black-box behavior can change for several reasons:
1) The API leaks implementation details and the implementation needs to change for some reason.
2) A new feature is added.
3) The domain changes. (typically rare)
4) There is an error in the implementation that leaks bad API responses.
API owners are motivated by management performance reviews. For 2-4, they will be given poor marks if they do not respond to defects and changing business demands, so they will implement the changes and update the Fake despite the additional cost.
Almost all bad APIs I've seen are examples of 1. This is a symptom of a bad design and probably made the Fake hard to write in the first place. In a system that hides no implementation details, the Fake is as complicated as the original system. I would hope that teams in this situation would come to the awareness of their fundamental problem and fix it since that would lower long-term costs for everyone.
A few years ago I worked with a team whose primary query API sent clients the entire storage record, which was a kind of state machine snapshot, consisting of dozens of poorly-defined, nested, high-cardinality fields. Clients generally only needed one or two pieces of data, but they had little guidance as what was public and unchanging and often picked an internal, implementation-specific field to use. The service owners didn't know what clients needed because they were internally focused. Imagine how hard that API would be to support with a Fake and how motivated that team would be to learn what the domain actually was and implement an API using that knowledge if a realistic Fake was mandatory.
Dave
p.s. In case you're curious, the team eventually did move to a better API. They came to realize how much their API constrained their implementation since any internal change was a potential hard-to-debug defect only caught in integration tests.
Thanks for your thorough reply Dave.
DeleteI've neglected to think about the many other factors at play in a significant API change; substantial rewrite of a fake may only play a minor role. But the motivation write a good API to make it easy to write fakes is at full force in both initial design and painful redesign of the API.
Daniel
Very nice post Dave
ReplyDeleteHi Dave - Thank you for writing out this post to minimize product-wide problems. In particular I was interested in Fakes as a solution to identify if an API is easy to use. I don't know if this would be the right place to inquire implementation details of Fakes but thought I would ask:
ReplyDelete1) Is Fakes a good way for controlling the underlying persisted data(test data/fixtures) which the actual service use to generate the responses?
2) And if so, Should Fakes also implement ways to insert and delete test data/fixtures?
> 1) Is Fakes a good way for controlling the underlying persisted data(test data/fixtures) which the actual service use to generate the responses?
DeleteI think the answer to your question is entirely dependent on your architecture. The only hard-and-fast definition of Fakes is that they are simplified implementations of the service useful for functional testing.
I work on a product now where the external API is basically a filesystem as web service, so a good Fake at that level would definitely not control an underlying persistent datastore but would most likely use an in-memory representation. I've worked on a product where there are multiple levels of bidirectional-interacting services, some of which touch the same underlying datastores or otherwise have side communication channels such that you can't write a Fake of service A without some way of propagating state changes (side-effects) to the Fake of service B.
It's probable that there are domains where such a complex, coupled architecture is inevitable. In those cases, the Fakes would have to have some communication channel, and it might be by talking to a lightweight common storage. However, I think it's likely that when confronted with the complication of writing and maintaining a web of interacting Fakes, teams will realize that there are better architectural alternatives (e.g. isolated microservices) in many cases. In either case, it's not the Fakes that make life difficult, they simply put the pain that is felt by the clients onto the service owners.
> 2) And if so, Should Fakes also implement ways to insert and delete test data/fixtures?
I think based on my answer to 1) the general answer is "no". For test fixtures using Fakes I expect the Fakes to provide seeded data for read-only services and for the test to generate the test data during the setup phase by making writes to read/write services' Fakes. I don't expect any of that data to reach a real database like Bigtable or MySQL because such heavy-weight dependencies are inappropriate for functional tests.
My approach to test data in end-to-end (Fake-less) tests is to generate it using the public APIs of the system, since they are least likely to change in ways that cause the data to violate the data stores' schema and most likely to generate realistic data (aside from thorny issues such as legacy/migrated data.)
Hi David I enjoyed your post, with fakes in your experience would you design them to support prod like response times delays or is production response delays a feature more suitable more a mock stub framework? I am asking this from a performance testing point of view. Thanks again for the post
ReplyDeleteHi John, I've helped SWEs add statistical delay strategies into fakes and that's worked fine. At Google we have a very standardized RPC system such that the often easier choice is to insert transparent, data-driven proxies that can fuzz timing as needed. In the case above the bespoke delay code was for performance testing a third-party, non-Google interaction so there was no existing infrastructure for inserting performance delays. If you have a common RPC mechanism, using dynamic proxies is the more general, lower-maintenance solution.
Delete