o Location: Offline (Room 201, N1 Building)

o Speaker : Yizheng Chen(University of Maryland)

■ Title: Benchmarking LLMs for Secure Code Generation

■ Abstract

Models (LLMs) have demonstrated promising capabilities in discovering and patching real-world security vulnerabilities. But how do we determine which LLM-based system performs best?

In this talk, I will explore the challenges of benchmarking LLMs for cyberdefense.

I will begin by presenting our work on evaluating LLMs’ ability to generate secure code. Notably, we find that results from prior code-generation benchmarks do not translate to LLMs’ secure coding performance in real-world software projects.

Next, I will discuss a key issue: memorization. LLMs may not be solving security problems from first principles but rather recalling secure solutions they have already seen. Finally, I will discuss future research directions in effectively evaluating

and improving LLMs for cybersecurity applications.

Models (LLM

■ Bio

Yizheng Chen is an Assistant Professor of Computer Science at the University of Maryland. Her research focuses on Large Language Models for Code Generation and AI for Security.

Her recent work PrimeVul has been used by Gemini 1.5 Pro for vulnerability detection evaluation. Previously, she received her Ph.D. in Computer Science from the Georgia Institute of Technology,

and was a postdoc at University of California, Berkeley and Columbia University. Her work has received an ACM CCS Best Paper Award Runner-up, a Google ASPIRE Award, and Top 10 Finalist of the CSAW Applied Research Competition.

She is a recipient of the Anita Borg Memorial Scholarship.

[Main research seminar] Professor Mark S. Ackerman from the University of Michigan, School of Information will give an in-person talk on sociotechnical approaches to computing. Specifically, he will be reflecting on his several decades of research helping pioneer the sociotechnical study and design of making computing systems useful for humans and organizations, drawing lessons for future research.
Date and time: February 7 (Friday) 10:30 AM - 12:00 PM (KST)
Location: N22 1st floor Room 103 (이민화홀)
Language: English
Host: Prof. Tom Steinberger

The memory wall has long been recognized as a critical challenge in high-performance systems, and it has recently become even more significant due to the exponential growth of machine learning model sizes. Meanwhile, recent advancements in interconnect technology, such as Compute Express Link (CXL), enable scalable memory system designs to address the memory capacity wall. Moreover, by offloading data and computation to CXL memory expanders to realize Near-Data Processing (NDP), the memory bandwidth wall can also be effectively mitigated. However, designing such a system should be done carefully, considering various design aspects that can affect the practicality of the solution.
In this talk, I will discuss key considerations and directions for building a practical NDP system architecture, including general-purpose computing, low-latency host communication, standard compliance, and cost-effectiveness. I will then present our recent work on an NDP architecture called Memory-Mapped NDP (M²NDP). M²NDP consists of two components: 1) Memory-Mapped Function (M²func), which enables low-latency host-device communication by addressing the overhead of conventional ring buffer-based task offloading, and 2) Memory-Mapped μthreading (M²μthread), a general-purpose, cost-effective NDP unit architecture that aims to maximize resource utilization by hybridizing CPU and GPU architectures. Finally, I will briefly outline future research directions based on the M²NDP architecture.

Gwangsun Kim is an Assistant Professor in the Department of Computer Science and Engineering at POSTECH. Previously, he was a Senior Research Engineer and Senior Performance Engineer at Arm Inc. He received the B.S. degrees in Electronic and Electrical Engineering and Computer Science and Engineering from POSTECH in 2010, and the M.S. and Ph.D. degrees in Computer Science from KAIST in 2012 and 2016, respectively. He has worked on various areas of computer architecture and systems, including memory systems, parallel architectures, GPU computing, systems for machine learning, near-data processing, networking, deep learning compiler, and simulation methodology. He is particularly interested in designing practical architectures for high-performance and scalable systems.

Despite significant progress in verifying protocols, services that implement distributed protocols , e.g., Chubby or Etcd, can exhibit safety bugs in production deployments. These bugs are often introduced by programmers when converting protocol descriptions into code. In this talk I will describe a new technique we have been developing to identify these bugs at runtime: Runtime Protocol Refinement Checking} (RPRC). RPRC systems observe a deployed service's runtime behavior and notify operators when this behavior evidences a protocol implementation bug, allowing operators to mitigate the bugs impact and developers to fix the bug. We have developed an algorithm for RPRC and implemented it in a system called Ellsberg that targets services that assume the asynchronous or partially synchronous model, and fail-stop failures. We designed Ellsberg so it makes no assumptions about how services are implemented, and requires no additional coordination or communication. We have used Ellsberg with three open source services: Etcd, Zookeeper and Redis Raft.

Aurojit Panda is an assistant professor in the Computer Science department at New York University working on systems and networking. He received his PhD in 2017 from UC Berkeley, where he was advised by Scott Shenker. He has received several awards, including a VMware Early Career Faculty Award, a Google Research Scholar Award, an NSF Career award, best paper awards at EuroSys, SIGCOMM and OSDI, and a EuroSys test of time award.