Theo d’Hondt (TH): Kate, you confirmed our suspicion that Java and C# developers do not really use OO programming.
Kate (KP): Yes, this is true. Maybe, this is not really needed in practice. We teach students design patterns in very simple cases, like circle and rectangle, but in practice things aren’t that clear. So my advice would be to address this in education.
This is why feedback on language use is so important, so understand what is actually used.
TH: But does that mean we should aim our languages at the amateurs and stop designing powerful stuff?
KP: Well, to be fair, we usually only study open source stuff, which might very well not represent realistic projects. For industrial software, this is different.
TH: Am I allowed to ask you whether you find more professional software at your current workplace (Google, Felienne)
KP: Yes, but I cannot comment furter
TH: You state that corpora are very hard to reuse and experiments are hard to reproduce. Even a corpus that is intended for reuse was not so easy to reuse. If we would be doing this in the bio sciences, they would be totally shocked, that is a life and death situation. You are very polite in your thesis, but you argue we are going about it the wrong way. Can you comment somewhat stringer?
KP: What surprised me the most is that for many corpora, you have to explicitly request access, which prevents use. Secondly, usually, when the student leaves the university, projects are abandoned. Often, project websites are not updated or even deleted. At MSR, what they do is they offer a dataset (the challenge) on which projects are ran. Maybe it is better as conference to release a few datasets that all papers (that apply) have to use.
TH: Why not ask people to deliver the data?
KP: Yes, as long as this is managed by a third-party.
TH: Is this the way it is done in biosciences?
KP: I have not experience in that field.
Tom Mens (TM): Why did you use SourceFourge as a datasource? These are typically small projects maintained by a few people. Would the results have been different if you had limited the results to larger projects? Did you see this effect in the data?
KP: We used SourceForge for historical reasons. In the beginning, we used used the available resources. With SF it was easy to identify projects we wanted to study and easily set up a tool chain. We did mitigate this effect by looking at serious projects only.
In this picture, you can see that the reference projects (the red dots) behave differently
TM: What kind of hypothesis would you define for this?
KP: For instance: The ratio of API method calls is higher for reference projects than for others.
TM: You created a better version of the Qualitas corpus. Has it already been used by the community?
KP: We refined it because we needed to be able to run and build the corpus ourselves. Some people have this need too, and one person has already used it.
TM: The community is aware of the issues with existing corpora, and when I mentioned that there is something like the Qualitas corpus. The answer was that this is a cleaned corpus, which impairs our results, because our tools need to be able to work with dirty realistic stuff. What is your comment on that?
KP: If I may continue to be blunt, this sounds more like an excuse. The quality of academic software will never be as high as professional software. So the first thing we need is to be able to do a fair comparison. If you look at what linguistics is doing, they do the same thing. They start with the raw data, but curate also. Knuth also removed broken punch cards from the data set, because it does not make sense to analyze them, they skew results.
Ralf Lammel (RL): You were making a fast movement from language use to APIs. In what sense do you consider APIs as languages?
KP: In my thesis I cite a call for papers from SLE where they consider all XML based languages an API. If Java contains XML, that can be considered a call.
RL: Can you think about a more complex example?
KP: I could say that all APIs are DSLs, but that is easy too. Certain methods of an API represent some aspects of a domain, and in that sense they cover a part pf a domain.
RL: What about language usage patterns?
KP: That is more about detecting patterns in the API than in the domain.
RL: Can you explain how your results for P3P generalize to more well-known small DSLs? Because no one in the room beside you and me know P3P
KP: Analysis of cloning can be helpful for any DSL. Snippets that occur often can be signs that a higher-level concept in the language in needed.
RL: P3P is not so commonly used anymore. Facebook, Twitter etc all have different privacy policies, expressed in their own language and system. What challenges do you see here?
KP: One approach is to measure the input options the users have (by crawling the HTML elements) this way we could come up with a general schema. Another way is to run content analysis on terms and agreements. P3P was just a way to simplify the long user agreements.
Chairman (C): Have I understood correctly that software programmers do not use them as intended?
KP: Well, to be precise, they do not use them as we think they should use them.
C: What is your advice? It seems that the languages are designed by people far away from the users. Would the situation be better if these people interacted more?
KP: Well, legend has it that Guido van Rossum designed Python when he was bored in the Christmas break, so that makes it a bit hard to study methodological design. I think, iterative design is the best, even though upgrading is expensive.
TH: I do not agree! This is the way that languages grow, for sure, but not how they should grow. Java 8 for instance, included Lambdas because the community wanted it, and now it is ruined into a Frankenstein which is a victim of its own success.
KP: No, my point was that for a certain period of time, a small testing environment is good. But for Java, that testing environment is now the whole world.
RL: May I also have an opinion? The problem is that they want to have backwards compatibility.
KP: Maybe instead of renaming from Java 1 to Java 2, we should rename to Quava or something, indicating it is a really new language.
TM: You have developed a tool for Java API analysis, would it be easy and feasible to make this tool into a tool for any API language?
KP: Once a project is loaded into Eclipse, you can implement your own tool, because you have access to all data. But I do not believe some much in very general tools and furthermore the community is not so interested in these tools.
TM: Maybe for a paper, but in your thesis there is certainly room for details on your tools.
TH: In a sense, your work is meta-empirical work, so I can imagine people are interested in this.
KP: I am not a hypocrite, once I started to work on this, I have uploaded all my data online, including for example all coding of the literature survey.
TH: Grounded theory says that hypotheses emerge from experiments. In order for the results to be valid, the experiments need to be reproducible.
KP: Well, strictly speaking, grounded theory is about qualitative analysis, not experiments. It was a good fit because it applies to an unclear field.
RL: Detour question: What are the things that Knuth is well known for? (the defense has to be at least 60 minutes :), Felienne)
RL: So you don’t like attribute grammars?
RL: You have two research problems: language usage and software corpora. Where do you see your contributions?
KP: In the area of language use I see my contribution as being generic. I was not driven by small subproblems as related work.