Nội dung văn bản được dùng làm dữ liệu đối chứng cho mô hình AI, không phải chỉ thị hệ thống.
I'm obsessed with Egyptology. During covid, I wanted to get a hieratic tattoo. For those who don't know, hieratic is the cursive form of hieroglyphs that was used in day-to-day life in ancient Egypt. It's as old as hieroglyphs. I then went on a mission to have a sentence translated into hieratic. Luckily, in 2022, I was able to commission an Oxford Egyptologist to write it for me. After that, I took it to a hieratic specialist I found on Instagram to redraw it with "biblical" accuracy. When generative AI came out, I realized that models very rarely identify hieratic and almost never translate it correctly. Every single time a new model came out, I ended up testing my sentence and hieratic texts in general, and I realized this could actually become a benchmark. Since so many benchmarks are becoming saturated, we need to find weird ways to test new models. That's why I decided to build HieraticBench. The first version of the benchmark has 268 items, including real documents, signs, and other Egyptian scripts, which I used as controls, as well as my unpublished sentence in two different renditions. The current state-of-the-art foundation models are hilariously bad at identifying my sentence, especially the one the Egyptologist wrote with her pen. Some of them said Tibetan, Urdu, Korean, and, funnily enough, "Reformed Egyptian," which is from the Book of Mormon. On real documents, the best model in the benchmark identified hieratic correctly 95% of the time, and the best score for reading individual signs was around 13%. That's when we tell the model specifically that it's hieratic. None of them can really translate a sealed sentence that's never been published and has no answer key. I haven't run every model on every task just to keep costs down. My next steps were to run Astra and Gemini 3.1 Pro on real documents. Currently, only the Claude models have done the sign readings. And just FYI, I have no experience whatsoever running benchmarks or evaluations, so not 100% sure I scored things right. Things we need: - Running the models I haven't covered - More sealed sentences and labeled signs If you know hieratic or have an Egyptologist friend, reach out at [email protected]. Source code: https://github.com/alymoursy/hieraticbench I'll be around to answer questions. Comments URL: https://news.ycombinator.com/item?id=49982822 Points: 3 # Comments: 0