How To Cook Spellcheckers, или Проверка орфографии — это просто
Моя прошлогодняя лекция, на русском 🙂
Моя прошлогодняя лекция, на русском 🙂
Редкая возможность поработать в успешной продуктовой компании, создавшей продукт мирового класса — то есть у нас, в Grammarly 🙂
Вакансия: Backend Engineer, relocation в Киев.
I was developing a small web server using Jetty and Apache Commons FileUpload component when I faced the problem with automated testing of the upload functionality. Jetty has a very nice framework to do tests: ServletTester and HttpTester.
ServletTester tester = new ServletTester(); tester.setContextPath("/"); tester.addServlet(TestingHandler.class, "/"); tester.start(); HttpTester request = new HttpTester(); HttpTester response = new HttpTester(); request.setMethod("GET"); request.setURI("/"); response.parse(tester.getResponses(request.generate())); assertEquals(200, response.getStatus());
My test had to upload a binary file to check the server-side processing, but I’ve found this is not an easy thing to do. I’ve realized that HttpTester class generate() method produces a String. Okay, let’s use BASE64 to encode the file! But this would not work, because FileUpload supports only «application/octet-stream» content type. Fortunately, FileUpload is an open source product, so I’ve just created a child class of HttpTester with my new generateBytes() method that did the trick. Basically, it is a slightly modified version of HttpTester.generate():
public static class HttpTesterBytes extends HttpTester { public HttpTesterBytes() { super(); setHeader("Host", "tester"); } public ByteArrayBuffer generateBytes() throws IOException { Buffer bb=new ByteArrayBuffer(32*1024 + (_genContent!=null?_genContent.length:0)); Buffer sb=new ByteArrayBuffer(4*1024); ByteArrayOutputStream2 out = new ByteArrayOutputStream2(); StreamEndPoint endp = new StreamEndPoint(null, out); HttpGenerator generator = new HttpGenerator(new SimpleBuffers(sb, bb),endp); if (_method!=null) { generator.setRequest(getMethod(),getURI()); if (_version==null) generator.setVersion(HttpVersions.HTTP_1_1_ORDINAL); else generator.setVersion(HttpVersions.CACHE.getOrdinal(HttpVersions.CACHE.lookup(_version))); generator.completeHeader(_fields,false); if (_genContent!=null) generator.addContent(new View(new ByteArrayBuffer(_genContent)),false); else if (_parsedContent!=null) generator.addContent(new ByteArrayBuffer(_parsedContent.toByteArray()),false); } generator.complete(); generator.flushBuffer(); return new ByteArrayBuffer(out.toByteArray()); } }
To prepare the bytes I’ve made a helper function:
public final static String CONTENT_BOUNDARY = "<--(*)(*)-->"; // Spaceship! /** * Get the binary representation of the file upload * @param fieldName field name to use in the form * @param filename file to load * @return bytes array with file written * @throws IOException */ public static byte[] getFileRequestContent(String fieldName, String filename, boolean finalize) throws IOException { String content = "--" + CONTENT_BOUNDARY + "\r\n" + "Content-Disposition: form-data; name=\"" + fieldName + "\"; filename=\"" + filename + "\"\r\n"+ "Content-Type: application/octet-stream;\r\n\r\n"; File f = new File(filename); byte[] bytes = Files.toByteArray(f); ByteArrayOutputStream stream = new ByteArrayOutputStream2(); stream.write(content.getBytes()); stream.write(bytes); String finish = "\r\n"; if (finalize) finish += "--" + CONTENT_BOUNDARY + "--\r\n\r\n"; stream.write(finish.getBytes()); stream.flush(); return stream.toByteArray(); }
And now how it looks in the test:
HttpTesterBytes request = new HttpTesterBytes(); HttpTester response = new HttpTester(); request.setMethod("POST"); request.setHeader("Content-Type", "multipart/form-data; boundary=" + CONTENT_BOUNDARY); request.setContentBytes(getFileRequestContent("data_file, "test/Nocturne.jpg")); response.parse(tester.getResponses(request.generateBytes()).array()); assertTrue(response.getMethod() == null); assertEquals(200, response.getStatus()); assertTrue(response.getContent().contains("cute!"));
double[] sx = {0.0, 1.0, 2.0, 3.0, 4.0}; double[] sy = {3.4, 5.2, 7, 8.8, 10.6}; double M = sx.length; double sumXY = 0, sumX = 0, sumY = 0, sumX2 = 0; for (int i = 0; i < M; i++) { sumX += sx[i]; sumX2 += sx[i] * sx[i]; sumY += sy[i]; sumXY += sx[i] * sy[i]; } double w1 = (M * sumXY - sumX * sumY) / (M * sumX2 - sumX * sumX); double w0 = 1 / M * (sumY - w1 * sumX); System.out.println("y = " + w1 + " * x + " + w0);