debug/elf: add SHT_GNU_VERDEF section parsing

[benbaker76]

I'm porting @brendangregg's eBPF profiling application profile.py to Golang. I'm attempting to use the debug/elf package for retrieving debug symbol info but have stumbled upon a limitation.

As a demonstration I've picked a library called libtiffxx.so as it's relatively small and clearly demontrates the issues I have with the package.

Let's take a look at the .gnu.version* sections contained in the library:

$ readelf -SW /usr/lib/x86_64-linux-gnu/libtiffxx.so | grep .gnu.version
  [ 6] .gnu.version      VERSYM          0000000000000882 000882 000036 02   A  4   0  2
  [ 7] .gnu.version_d    VERDEF          00000000000008b8 0008b8 000038 00   A  5   2  8
  [ 8] .gnu.version_r    VERNEED         00000000000008f0 0008f0 000090 00   A  5   3  8

There's important debug symbol information contained in the .gnu.version_d (SHT_GNU_VERDEF) section which is currently not parsed by the package. Let's look at the last four functions in the .dynsym section.

$ readelf -sW /usr/lib/x86_64-linux-gnu/libtiffxx.so | tail -4
    23: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND _ZNSt8ios_base4InitD1Ev@GLIBCXX_3.4 (3)
    24: 0000000000000000     0 OBJECT  GLOBAL DEFAULT  ABS LIBTIFFXX_4.0
    25: 0000000000001890   169 FUNC    GLOBAL DEFAULT   15 _Z14TIFFStreamOpenPKcPSo@@LIBTIFFXX_4.0
    26: 0000000000001940    19 FUNC    GLOBAL DEFAULT   15 _Z14TIFFStreamOpenPKcPSi@@LIBTIFFXX_4.0

The debug/elf package can retreive the function name, but what about the @ or @@, the library name after it or even the number in the brackets? In my proposal and my current implementation I provide a way to access this important debug symbol information.

If we take a look at the dynamic symbol output of these four functions in the current implementation of debug/elf we have the following output:

		Symbol{
			Name:     "_ZNSt8ios_base4InitD1Ev",
			Info:     0x12,
			Other:    0x0,
			Section:  0x0,
			Value:    0x0,
			Size:     0x0,
			Version:  "GLIBCXX_3.4",
			Library:  "libstdc++.so.6",
		},
		Symbol{
			Name:     "LIBTIFFXX_4.0",
			Info:     0x11,
			Other:    0x0,
			Section:  0xFFF1,
			Value:    0x0,
			Size:     0x0,
		},
		Symbol{
			Name:     "_Z14TIFFStreamOpenPKcPSo",
			Info:     0x12,
			Other:    0x0,
			Section:  0xF,
			Value:    0x1890,
			Size:     0xA9,
		},
		Symbol{
			Name:     "_Z14TIFFStreamOpenPKcPSi",
			Info:     0x12,
			Other:    0x0,
			Section:  0xF,
			Value:    0x1940,
			Size:     0x13,
		},

In my new implemenation we get this output:

		Symbol{
			Name:     "_ZNSt8ios_base4InitD1Ev",
			Info:     0x12,
			Other:    0x0,
			Section:  0x0,
			Value:    0x0,
			Size:     0x0,
			Version:  "GLIBCXX_3.4",
			Library:  "libstdc++.so.6",
			VerInfo:  0x1,
			VerOther: 0x3,
		},
		Symbol{
			Name:     "LIBTIFFXX_4.0",
			Info:     0x11,
			Other:    0x0,
			Section:  0xFFF1,
			Value:    0x0,
			Size:     0x0,
			Version:  "LIBTIFFXX_4.0",
			Library:  "",
			VerInfo:  0x2,
			VerOther: 0x1,
		},
		Symbol{
			Name:     "_Z14TIFFStreamOpenPKcPSo",
			Info:     0x12,
			Other:    0x0,
			Section:  0xF,
			Value:    0x1890,
			Size:     0xA9,
			Version:  "LIBTIFFXX_4.0",
			Library:  "",
			VerInfo:  0x2,
			VerOther: 0x1,
		},
		Symbol{
			Name:     "_Z14TIFFStreamOpenPKcPSi",
			Info:     0x12,
			Other:    0x0,
			Section:  0xF,
			Value:    0x1940,
			Size:     0x13,
			Version:  "LIBTIFFXX_4.0",
			Library:  "",
			VerInfo:  0x2,
			VerOther: 0x1,
		},

We now have the Version entry as is often retrieved from SHT_GNU_VERNEED but in the last three entries can only be obtained from the SHT_GNU_VERDEF section. We also have two new Symbol members called VerInfo and VerOther.

So what are these new values and where do they come from? Using binutil's readelf.c as a reference we can find out where this data is located and how to extract it.

VerInfo in this case refers to the number in the brackets. In the case of _ZNSt8ios_base4InitD1Ev@GLIBCXX_3.4 (3) the number 3 comes from VerOther which is the version number. If it's 1 we don't display it in the brackets. The single @ comes from VerInfo which can be one of three values:

// Versioned symbol info
const (
	SymUndefined byte = 0
	SymHidden    byte = 1
	SymPublic    byte = 2
)

If the value is SymHidden it will be @ and if it's SymPublic it will be @@. This is taken from readelf.c

So I have implemented the code to do the necessary parsing of the SHT_GNU_VERDEF section and am writing this proposal so I can submit my changes to Gerrit for review by the Golang team.

Currently I have made changes to the following files in src/debug/elf:

• Change file_test.go
• Change file.go
• Change symbols_test.go
• Added testdata/libtiffxx.so (should be renamed?)

As I mentioned I use libtiffxx.so as my testdata ELF binary because it's relatively small (14.4 kB) and it demonstrates the new data from which my new branch can parse.

My changes currently pass all tests except for the API check

##### API check
+pkg debug/elf, const SymHidden = 1
+pkg debug/elf, const SymHidden uint8
+pkg debug/elf, const SymPublic = 2
+pkg debug/elf, const SymPublic uint8
+pkg debug/elf, const SymUndefined = 0
+pkg debug/elf, const SymUndefined uint8
+pkg debug/elf, type ImportedSymbol struct, VerInfo uint8
+pkg debug/elf, type ImportedSymbol struct, VerOther uint8
+pkg debug/elf, type Symbol struct, VerInfo uint8
+pkg debug/elf, type Symbol struct, VerOther uint8
--- FAIL: TestCheck (26.62s)
    main_test.go:184: API differences found

Thanks for taking the time to review my proposal.

[rsc]

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

[rsc]

This API diff is nice and small and seems reasonable for the benefit:

##### API check
+pkg debug/elf, const SymHidden = 1
+pkg debug/elf, const SymHidden uint8
+pkg debug/elf, const SymPublic = 2
+pkg debug/elf, const SymPublic uint8
+pkg debug/elf, const SymUndefined = 0
+pkg debug/elf, const SymUndefined uint8
+pkg debug/elf, type ImportedSymbol struct, VerInfo uint8
+pkg debug/elf, type ImportedSymbol struct, VerOther uint8
+pkg debug/elf, type Symbol struct, VerInfo uint8
+pkg debug/elf, type Symbol struct, VerOther uint8

Are there any remaining concerns about this proposal?

[cherrymui]

Could you explain what the meanings of VerInfo and VerOther fields are? Thanks.

[benbaker76]

They come from readelf.c

If we take a look at some output from it:

$ readelf -sW /usr/lib/x86_64-linux-gnu/libtiffxx.so | tail -4
    23: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND _ZNSt8ios_base4InitD1Ev@GLIBCXX_3.4 (3)
    24: 0000000000000000     0 OBJECT  GLOBAL DEFAULT  ABS LIBTIFFXX_4.0
    25: 0000000000001890   169 FUNC    GLOBAL DEFAULT   15 _Z14TIFFStreamOpenPKcPSo@@LIBTIFFXX_4.0
    26: 0000000000001940    19 FUNC    GLOBAL DEFAULT   15 _Z14TIFFStreamOpenPKcPSi@@LIBTIFFXX_4.0

VerInfo is synonymous with sym_info and VerOther with vna_other.

// Versioned symbol info
const (
	SymUndefined byte = 0
	SymHidden    byte = 1
	SymPublic    byte = 2
)

Comes from:

https://github.com/bminor/binutils-gdb/blob/master/binutils/readelf.c#L363

/* Versioned symbol info.  */
enum versioned_symbol_info
{
  symbol_undefined,
  symbol_hidden,
  symbol_public
};

https://github.com/bminor/binutils-gdb/blob/master/binutils/readelf.c#L13764

  if (version_string)
    {
      if (sym_info == symbol_undefined)
	printf ("@%s (%d)", version_string, vna_other);
      else
	printf (sym_info == symbol_hidden ? "@%s" : "@@%s",
		version_string);
    }

In the case of _ZNSt8ios_base4InitD1Ev@GLIBCXX_3.4 (3) vna_other is 3 and sym_info == symbol_hidden.

So in the actual implementation in go/src/debug/elf/file.go we have:

// Versioned symbol info
const (
	SymUndefined byte = 0
	SymHidden    byte = 1
	SymPublic    byte = 2
)

type verdef struct {
	Other uint16
	Name  string
}

type verneed struct {
	File  string
	Other uint16
	Name  string
}

// gnuVersionInit parses the GNU version tables
// for use by calls to gnuVersion.
func (f *File) gnuVersionInit(str []byte) bool {
	// Accumulate verdef information.
	var def []verdef
	vd := f.SectionByType(SHT_GNU_VERDEF)
	if vd != nil {
		d, _ := vd.Data()

		i := 0
		for {
			if i+20 > len(d) {
				break
			}
			other := f.ByteOrder.Uint16(d[i : i+2])
			//flags := f.ByteOrder.Uint16(d[i+2 : i+4])
			ndx := int(f.ByteOrder.Uint16(d[i+4 : i+6]))
			cnt := f.ByteOrder.Uint16(d[i+6 : i+8])
			// hash := f.ByteOrder.Uint32(d[i+8 : i+12])
			aux := f.ByteOrder.Uint32(d[i+12 : i+16])
			next := f.ByteOrder.Uint32(d[i+16 : i+20])

			var name string
			j := i + int(aux)
			for c := 0; c < int(cnt); c++ {
				if j+8 > len(d) {
					break
				}
				vname := f.ByteOrder.Uint32(d[j : j+4])
				vnext := f.ByteOrder.Uint32(d[j+4 : j+8])
				name, _ = getString(str, int(vname))

				if c == 0 {
					if ndx >= len(def) {
						a := make([]verdef, ndx)
						copy(a, def)
						def = a
					}

					def[ndx-1] = verdef{other, name}
				}

				j += int(vnext)
			}

			if next == 0 {
				break
			}
			i += int(next)
		}
	}

	// Accumulate verneed information.
	var need []verneed
	vn := f.SectionByType(SHT_GNU_VERNEED)
	if vn != nil {
		d, _ := vn.Data()

		i := 0
		for {
			if i+16 > len(d) {
				break
			}
			vers := f.ByteOrder.Uint16(d[i : i+2])
			if vers != 1 {
				break
			}
			cnt := f.ByteOrder.Uint16(d[i+2 : i+4])
			fileoff := f.ByteOrder.Uint32(d[i+4 : i+8])
			aux := f.ByteOrder.Uint32(d[i+8 : i+12])
			next := f.ByteOrder.Uint32(d[i+12 : i+16])
			file, _ := getString(str, int(fileoff))

			var name string
			j := i + int(aux)
			for c := 0; c < int(cnt); c++ {
				if j+16 > len(d) {
					break
				}
				// hash := f.ByteOrder.Uint32(d[j : j+4])
				// flags := f.ByteOrder.Uint16(d[j+4 : j+6])
				other := f.ByteOrder.Uint16(d[j+6 : j+8])
				nameoff := f.ByteOrder.Uint32(d[j+8 : j+12])
				next := f.ByteOrder.Uint32(d[j+12 : j+16])
				name, _ = getString(str, int(nameoff))
				ndx := int(other)

				if ndx >= len(need) {
					a := make([]verneed, 2*(ndx+1))
					copy(a, need)
					need = a
				}

				need[ndx] = verneed{file, other, name}
				if next == 0 {
					break
				}
				j += int(next)
			}

			if next == 0 {
				break
			}
			i += int(next)
		}
	}

	// Versym parallels symbol table, indexing into verneed.
	vs := f.SectionByType(SHT_GNU_VERSYM)
	if vs == nil {
		return false
	}
	d, _ := vs.Data()

	f.gnuVerdef = def
	f.gnuNeed = need
	f.gnuVersym = d
	return true
}

// gnuVersion adds Library and Version information to sym,
// which came from offset i of the symbol table.
func (f *File) gnuVersion(i int) (library string, version string, info byte, other byte) {
	// Each entry is two bytes; skip undef entry at beginning.
	i = (i + 1) * 2
	if i >= len(f.gnuVersym) {
		return
	}
	s := f.gnuVersym[i:]
	if len(s) < 2 {
		return
	}
	j := int(f.ByteOrder.Uint16(s))
	if j >= 2 && j < len(f.gnuNeed) {
		n := &f.gnuNeed[j]
		if n.File != "" {
			return n.File, n.Name, SymHidden, byte(n.Other)
		}
	}

	var verInfo byte = SymUndefined

	if j&0x8000 != 0 {
		verInfo = SymHidden
	} else {
		verInfo = SymPublic
	}

	j = (j & 0x7fff) - 1
	if j >= 1 && j < len(f.gnuVerdef) {
		v := &f.gnuVerdef[j]
		return "", v.Name, verInfo, byte(v.Other)
	}

	return
}

The 0x8000 and 0x7fff from above are from:

https://github.com/bminor/binutils-gdb/blob/master/include/elf/common.h#L1306

/* This flag appears in a Versym structure.  It means that the symbol
   is hidden, and is only visible with an explicit version number.
   This is a GNU extension.  */

#define VERSYM_HIDDEN		0x8000

/* This is the mask for the rest of the Versym information.  */

#define VERSYM_VERSION		0x7fff

[rsc]

It does seem strange for Sym* constants to go into a VerInfo field. Perhaps the names should be made closer?

[benbaker76]

It does seem strange for Sym* constants to go into a VerInfo field. Perhaps the names should be made closer?

readelf.c calls the enum versioned_symbol_info. I did consider prefixing with VerSym* but it seemed overly verbose. For example VerSymUndefined. The readelf.c enums are prefixed with symbol_* so it was in reference to that.

[benbaker76]

// Versioned symbol info
const (
	VerSymUndefined byte = 0
	VerSymHidden    byte = 1
	VerSymPublic    byte = 2
)

@rsc would you prefer something like this?

[ianlancetaylor]

I know you know all this, but just to reprise:

There are three sections that contain version information: SHT_GNU_VERSYM (.gnu.version), SHT_GNU_VERDEF (.gnu.version_d), and SHT_GNU_VERNEED (.gnu.version_r).

The SHT_GNU_VERSYM section contains a 16-bit value for each symbol in the symbol table. The high bit (VERSYM_HIDDEN) is set if the symbol has hidden visibility, meaning that some other executable or dynamic library can only refer to it using an explicit version number. The other 15 bits may be 0 (VER_NDX_LOCAL), meaning that the symbol is not visible outside the object, or 1 (VER_NDX_GLOBAL), meaning that the symbol has no version, or some larger value which is an index into the list of versions.

The SHT_GNU_VERDEF section defines versions. The sh_info field of the section header is the number of versions. Each version has flags, an index, a name, and a list of versions that it depends on.

The SHT_GNU_VERNEED section defines versions that are required from other dynamic objects. The sh_info field of the section header is the number of entries. Each entry refers to some other dynamic object, and has a list of requirements. Each requirement is a flag and a version name.

The flags in SHT_GNU_VERDEF and SHT_GNU_VERNEED sections are VER_FLG_BASE for the base revision, and VER_FLG_WEAK and VER_FLG_INFO which are pretty esoteric.

Anyhow, the point is, the part of the version information that is symbol-specific is the SHT_GNU_VERSYM section. We've already locked ourselves into representing that as the Version and Library fields of Symbol. We currently ignore the symbol-specific VERSYM_HIDDEN flag, and we should fix that. But I don't see a reason to express the other version information as fields of Symbol. In particular a VerOther field doesn't make sense to me; that's a property of a version, not a symbol, and I don't see a reason to duplicate it across all the symbols.

So how about:

Add a VersionIndex int32 field to Symbol set from the entry in the SHT_GNU_VERSYM section. This will be set by DynamicSymbols. If there is no SHT_GNU_VERSYM section, the field will be set to -1 for all symbols. On a 64-bit system we can squeeze this field in without increasing the size of Symbol, by putting after Other.

Add a (*File).DynamicVersions method that returns version information. It will return []DynamicVersion.

// DynamicVersion is a version defined by a dynamic object.
type DynamicVersion struct {
    Version uint16 // Version of data structure.
    Flags DynamicVersionFlags
    Index uint16
    Name string
    Deps []string
}

We define flags for the DynamicVersion Flags field.

type DynamicVersionFlags uint16
const (
    VER_FLG_BASE DynamicVersionFlags = 1
    VER_FLG_WEAK DynamicVersionFlags = 2
    VER_FLG_INFO DynamicVersionFlags = 4
)

Add a (*File).DynamicVersionNeeds method that returns dependencies. It will return []DynamicVersionNeed.

type DynamicVersionNeed struct {
    Version uint16 // version of data structure
    Name string // shared library  name
    Needs []DynamicVersionDep
}

type DynamicVersionDep struct {
    Flags DynamicVersionFlags
    Other uint16
    Dep string // Name of required version
}

I think this approach will let us extract all the information from the dynamic object, and let people write tools to do whatever they want.

[benbaker76]

@ianlancetaylor thanks for your feedback and additional info. While your approach does appear to work I don't see the need to separate the version data from Symbol and ImportedSymbol structs. My proposal adds two additional bytes and re-uses Version and Library members without affecting any existing data.

To be sure I wrote a test which will compare the output of readelf -sW with all the libraries located in /usr/lib/x86_64-linux-gnu of Ubuntu 23.10.

//go:build linux

package main

import (
	"bufio"
	"debug/elf"
	"fmt"
	"os"
	"os/exec"
	"strconv"
	"strings"
	"testing"
)

type SymbolInfo struct {
	Num   int
	Value string
	Size  string
	Type  string
	Bind  string
	Vis   string
	Ndx   string
	Name  string
}

func TestSymbols(t *testing.T) {
	folderPath := "/usr/lib/x86_64-linux-gnu"
	searchPattern := ".so"

	files, err := searchFiles(folderPath, searchPattern)
	if err != nil {
		t.Errorf("Error %v", err)
		return
	}

	t.Logf("Found %d files", len(files))

	for _, file := range files {
		symbols, err := parseSymbols(file)
		if err != nil {
			continue
		}

		f, err := elf.Open(file)

		if err != nil {
			continue
		}

		var checkSymbols = make(map[string][]elf.Symbol)

		checkSymbols[".dynsym"], _ = f.DynamicSymbols()
		checkSymbols[".symtab"], _ = f.Symbols()

		for tableName, tableSymbols := range symbols {
			if len(tableSymbols) != len(checkSymbols[tableName]) {
				t.Errorf("expected %d got %d\n", len(tableSymbols), len(checkSymbols[tableName]))
			}

			for index, symbol := range tableSymbols {
				symbolCheck := checkSymbols[tableName][index]

				value := fmt.Sprintf("%016x", symbolCheck.Value)
				size := fmt.Sprintf("%d", symbolCheck.Size)
				if strings.HasPrefix(symbol.Size, "0x") {
					size = fmt.Sprintf("0x%x", symbolCheck.Size)
				}
				type_ := parseType(int(symbolCheck.Info & 0xf))
				bind := parseBind(int(symbolCheck.Info >> 4))
				vis := parseVisibility(int(symbolCheck.Other))
				ndx := parseNdx(int(symbolCheck.Section))
				name := getSymbolName(symbolCheck, AddLibrary, AddVersion)

				if value != symbol.Value {
					t.Errorf("expected %v got %v\n", symbol.Value, value)
				}
				if size != symbol.Size {
					t.Errorf("expected %x got %x\n", symbol.Size, size)
				}
				if type_ != symbol.Type {
					t.Errorf("expected %v got %v\n", symbol.Type, type_)
				}
				if bind != symbol.Bind {
					t.Errorf("expected %x got %v\n", symbol.Bind, bind)
				}
				if vis != symbol.Vis {
					t.Errorf("expected %v got %v\n", symbol.Vis, vis)
				}
				if ndx != symbol.Ndx {
					t.Errorf("expected %v got %v\n", symbol.Ndx, ndx)
				}
				if name != symbol.Name {
					t.Errorf("expected %v got %v\n", symbol.Name, name)
				}
			}
		}
	}
}

func searchFiles(folderPath, searchPattern string) ([]string, error) {
	var files []string

	file, err := os.Open(folderPath)
	if err != nil {
		return nil, err
	}
	defer file.Close()

	fileInfos, err := file.Readdir(-1)
	if err != nil {
		return nil, err
	}

	for _, fileInfo := range fileInfos {
		if !fileInfo.IsDir() && strings.Contains(fileInfo.Name(), searchPattern) {
			files = append(files, folderPath+"/"+fileInfo.Name())
		}
	}

	return files, nil
}

func parseSymbols(filePath string) (map[string][]SymbolInfo, error) {
	cmd := exec.Command("readelf", "-sW", filePath)
	output, err := cmd.CombinedOutput()
	if err != nil {
		return nil, err
	}

	scanner := bufio.NewScanner(strings.NewReader(string(output)))
	symbols := make(map[string][]SymbolInfo)
	var tableName string

	for scanner.Scan() {
		line := scanner.Text()

		if strings.HasPrefix(line, "Symbol table") {
			startPos := strings.Index(line, "'")
			endPos := strings.LastIndex(line, "'")
			tableName = line[startPos+1 : endPos]
			continue
		}

		if strings.HasPrefix(line, "   Num:") {
			scanner.Scan()
			continue
		}

		fields := strings.Fields(line)
		if len(fields) < 7 {
			continue
		}

		name := ""

		if len(fields) == 8 {
			name = fields[7]
		}

		symbol := SymbolInfo{
			Num:   len(symbols[tableName]),
			Value: fields[1],
			Size:  fields[2],
			Type:  fields[3],
			Bind:  fields[4],
			Vis:   fields[5],
			Ndx:   fields[6],
			Name:  name,
		}

		if len(fields) == 9 {
			symbol.Name = fields[7] + " " + fields[8]
		}

		symbols[tableName] = append(symbols[tableName], symbol)
	}

	return symbols, nil
}

func parseType(typeInt int) string {
	typeTable := map[int]string{
		0:  "NOTYPE",
		1:  "OBJECT",
		2:  "FUNC",
		3:  "SECTION",
		4:  "FILE",
		5:  "COMMON",
		6:  "TLS",
		10: "IFUNC",
		13: "LOPROC",
		15: "HIPROC",
	}

	return typeTable[typeInt]
}

func parseBind(bindInt int) string {
	bindTable := map[int]string{
		0:  "LOCAL",
		1:  "GLOBAL",
		2:  "WEAK",
		10: "UNIQUE",
		11: "UNIQUE",
		12: "UNIQUE",
	}

	return bindTable[bindInt]
}

func parseVisibility(visInt int) string {
	visTable := map[int]string{
		0: "DEFAULT",
		1: "INTERNAL",
		2: "HIDDEN",
		3: "PROTECTED",
	}

	return visTable[visInt]
}

func parseNdx(ndxInt int) string {
	ndxTable := map[int]string{
		0:      "UND",
		0xff00: "BEFORE",
		0xff01: "AFTER",
		0xff1f: "HIPROC",
		0xfff1: "ABS",
		0xfff2: "COMMON",
		0xffff: "HIRESERVE",
	}

	if val, ok := ndxTable[ndxInt]; ok {
		return val
	}

	return strconv.FormatInt(int64(ndxInt), 10)
}

$ go test ./cmd/profile -v
=== RUN   TestSymbols
    main_test.go:37: Found 3979 files
--- PASS: TestSymbols (40.48s)
PASS
ok      github.com/benbaker76/libbpf-tools/cmd/profile  40.488s