Evolution
Framework growthe
New domains
Bug fixes
The conflict
o o
/
o
Business models
Razors vs blades
Source
Documentation
Testing
How to test abstract classes?
[Make concrete]
What’s a test case ? [an application]
Software engineering of Frameworks
Iteration
Evolution
Domains
Bug fixes
Conflicts
Business model
Razor vs blades
Source or not
Documentation
Testing
Abstract classes
Test case = application
Software Engineering of Frameworks
Iteration
Evolution
Domains
Bug fixes
Conflicts
Biz model
Razor vs blades
Source
Documentation
Testing
Abstract classes
Test case = app
Iteration and Evolution
Frameworks don’t spring up in one piece “by unaided thought alone.” They develop through a reflective process, reflecting the tension between frameworks and their application.
< – gives ideas to –
Framework Application
– supports –>
As the framework is used for applications, the applications will show patterns of use that can be incorporated back into the framework.
This proceeds over time, and is a necessary part of framework development. A framework is best generalized from several applications.
A framework evolves for the “usual” software engineering reasons: to adapt to new environments, to enhance functionality, to repair defects.
What happens when the framework is changed? The problem is that not everybody will want to use the new version:
* the old version is used in an application that is already deployed
* the new version has new bugs
* it takes work to update.
For example, consider our first web site manager. Suppose there is version 1, which uses the Graph class that has a bug in it, and a framework. Suppose it grows to version 2, where the bug is fixed and a plugin architecture is used. The framework user of version 1 would like to see the bug fixed, but may not want to rewrite their part as a plugin.
The biggest problems often come because of bugs. Version 1 has certain bugs, which version 2 fixes, but version 2 changes other things. Customers of version 1 may want a bug-free version 1 (which never existed). It’s hard to do this from an engineering/management perspective.
Managing this usually requires tool support, from a source code control system. This lets us run multiple versions of software:
v1.0 – 1.1 – 1.2
|
v2.0 – 2.1 2.2 2.3
(where does bug fix go?)
This is a lot of work, and easy to get wrong. For example, suppose a key data structure changes from v1.0 to v2.0; how do we match up the bug fixes?
Business Models
Frameworks cost time and money to develop. How do you ensure the result justifies the investment?
The key consideration is that the cost of the framework should be amortized over a number of applications, because the benefits accrue to several applications.
Some authors (ref) have advocated a split between framework developers and application developers (some recommend two or three teams). The framework authors try to provide frameworks that the application team wants.
What do you sell when you “sell” your framework? Usually, it’s a license to use.
Do you sell source code or not? Some shops don’t want to depend on you being around to fix the bugs, or they need to see how things work.
Are you selling razors or blades? Consider web servers – there are perfectly good servers available for free (e.g., Apache). Many people can give away the web server (the razor) and sell things that use it (the blades). All the money’s in selling the blades. (This may be services as well as actual products.)
Testing
Framework level = app
Package level
Abstract class
Class
Method
Testing a framework involves concerns at a number of levels:
* Framework and its applications
* [Package Level]
* Abstract Class
* Concrete Class
* Subclasses
* Methods
* [Interface]
In some sense, the test case for a framework is an application.
Testing attempts to “break” the software: do something documented to work one way, but the system performs another way.
Framework testing:
The hardest part is that the test case for a framework is an application (a use of the framework). To test, we will build an application and test the framework through the application.
This gives you a chance to do “system level” tests of the framework and its documentation. Your documentation should have a minimal example of using the framework, and it should work as advertised.
[Key] If it’s not defined, it can’t be tested.
Class Level testing:
In object-oriented systems, we often can test classes bottom-up. It is sometimes feasible to develop a testing package for each class. When you’re lucky, you can test each one in isolation.
One special problem you face is in dealing with abstract classes. Since the point of the abstract class is to omit things (deferred to subclasses), the only way to test them is to subclass them with a concrete class, and test the concrete class.
You may introduce new concrete classes not part of the framework.
The Design by Contract constraints will help in this testing.
It’s hard to test a class in isolation. You often need to test the client too. If there are problems, it can be hard to tell if it’s the class or the client.
[Demo?]
* Create an instance. Test visible aspects of the initial configuration.
* Call modifiers. After modification, use accessors to verify configuration.
* Verify each method, each visible variable.
* Work from public to protected to private.
It’s tricky to test private variables or methods. Recall that in Java, items in the same package have access to private values. You might introduce a test class into the package to do this. You need to document the dependency. Perhaps you can use an inner class. Some classes will just require scaffolding that may need to be removed later.
It’s worth applying some intelligence in the testing – if a private variable is merely behind simple public getter/setter methods, simply testing the public methods is enough.
Subclass:
Test other relations
Decouple
Testing Methods:
*Write test cases that will stress the method. There are various kinds of coverage you can strive for:
+ all statements executed
+ all branches tried both ways
+ all methods excecuted
An effective way is to try corner cases. For integers, this might be a large negative number, -1, 0, 1, and a large positive number.
Stress Testing
In addition to a straightforward client, it is sometimes helpful to stress test a service. For a stress test, you want a client that can call the class many times. This can be tricky to write. For example, a stack class can’t do a random sequence of pushes and pops (without violating preconditions). The client would have to be smart enough to pop only there are values on the stack.
Ideally, you have some kind of oracle. An oracle is a magic box that tells you if the object under test is correct. In the worst case, the oracle is a re-implementation of the service. In other cases, your oracle isn’t all-knowing.
Example
TestingStack extends Vector {
public Stack()
public boolean empty()
public synchronized Object peek()
public synchronized Object pop()
public Object push(Object o)
public synchronized int search (Object o) // dist from top; -1=not present
}
0 == 1 == many
First, look at the modifiers only. The accessors should have no effect.
(new Stack()).empty() => true
s.peek() => error
s.pop() => error
s.push(o1) => s’
s.empty(s.push(o1)) => false
s.peek(s.push(o1)) => o1
etc.
Building an Oracle
When we test, we need to decide if the test passes or fails.
In simple cases, we just compute the answer by hand and use a one-shot oracle, e.g.,
empty(pop(push(stack))) = true
The problem with this is that we have to compute many answers separately.
Sometimes, there’s an alternative. We may have a separate implementation. For example, the JDK 1.2 collection classes provide multiple implementations of things like Set. So if we put the same operation to both sets, then we should get the same answer. If they don’t agree, there’s an error somewhere.
What can go wrong?
* Non-determinism. For example, sets might provide Enumerations in different orders.
* Oracles can share failure modes. The same people might have written both implementations, or there can be ambiguities in the specification.
* It’s hard to find an oracle.
An alternative is a reduced oracle. In some cases, we can make a “projection” of an oracle that will test only part of the state.
For example, a stack oracle might track the stack height, but not the values in it. After a series of operations, we can pop all values, and test that the stack is empty when the oracle’s count is 0.
Another example might be a data processing program. It might count records read and records written, and raise a flag if they don’t match.
The simplest oracle is “no crash” – there’s no explicit oracle, we just try a lot of operations and make sure the service doesn’t crash.
Once we have an oracle, we can do a stress test:
Stack stack = new Stack();
for (int i = 0; i < numiters; i++) {
float f = random(0,1);
if (f < 0.4) {
if (!stack.empty()) {
value = stack.peek();
value2 = stack.pop();
if (value != value2) error();
}
} else {
stack.push(value3);
}
}