Austin Liu austin362667

和一位朋友討論 LLM evaluation (成效評估) 和其一些 benchmark (比較基準) datasets 的相關問題

因為面對語言類型的任務通常沒有絕對的正確或錯誤，所以我們會設計一些資料來測驗 LLM 的能力，這些 benchmark datasets 只能作為驗證模型能力的某種 proxy。而且各種資料集有各自專精的領域，類型包羅萬象，諸如：邏輯型、情緒型、翻譯、程式碼、數學解題、常識推理等等族繁不及備載。
以 BBH 基準資料集來說：
- 文字輸入: False or not ( True ) and False is
我們會期望模型文字輸出: False

Why kernel code should use READ_ONCE and WRITE_ONCE for shared memory accesses

There are several reasons to use at least READ_ONCE and WRITE_ONCE for all concurrent non-read-only shared memory accesses:

	use std::{cmp::max, cmp::min, error::Error};
	use plotters::prelude::*;
	use ocl::{ProQue, Buffer, MemFlags};

	// dims = GL (gnum_threads + l)
	// OpenCL CUDA HIP
	// G cores (get_group_id, blockIdx, __ockl_get_group_id, threadgroup_position_in_grid)
	// L threads (get_local_id, threadIdx, __ockl_get_local_id, thread_position_in_threadgroup)

	// 128 cores, 1 thread each

	import numpy as np
	import torch
	from typing import List, Dict, Optional
	from dataclasses import dataclass
	from transformers import AutoTokenizer, AutoModelForCausalLM

	class HuggingFaceLLM:
	def __init__(self, model_name: str = "HuggingFaceTB/SmolLM2-135M-Instruct", device: str = 'mps'):
	self.tokenizer = AutoTokenizer.from_pretrained(model_name, device_map=device)
	self.model = AutoModelForCausalLM.from_pretrained(model_name, device_map=device)

	// WARNING: THIS CODE IS A HACKY PROTOTYPE FOR TOY PURPOSES ONLY.
	// CHOOSING TO COPY THIS CODE INTO YOUR RUST PROGRAM IS WILDLY UNSOUND.
	//
	// ping me (davidhewitt) on the PyO3 issue tracker and we can figure out
	// how to do this safely.

	use std::collections::{HashMap, hash_map::Iter as MapIter};

	// pyclass 1
	pub struct Class {

	module Main where
	-- WIP: Haven't read ch6, ch8 in [Learn You a Haskell for Great Good].

	-- 0.
	penultimate xs = last (init xs)
	-- 1.
	antepenultimate xs = penultimate (init xs)
	-- 2.
	-- `shiftLeft` is a helper function for `rotateLeft`
	shiftLeft [] = []

	def sim(ta, tb):
	a_t = enc.encode(ta)
	b_t = enc.encode(tb)
	a_te = np.array(model.transformer.wte.weight[a_t].tolist())
	b_te = np.array(model.transformer.wte.weight[b_t].tolist())
	a_pe = np.array(model.transformer.wpe.weight[[i for i in range(len(a_t))]].tolist())
	b_pe = np.array(model.transformer.wpe.weight[[i for i in range(len(b_t))]].tolist())
	x = np.add.reduce(a_te+a_pe)
	y = np.add.reduce(b_te+b_pe)

	openapi: 3.0.0
	info:
	title: WolframAlpha API
	version: 1.0.0
	servers:
	- url: https://api.wolframalpha.com/v1
	paths:
	/result:
	get:
	summary: Returns a simple answer from WolframAlpha

	// Copyright 2023 Optiver Asia Pacific Pty. Ltd.
	//
	// This file is part of Ready Trader Go.
	//
	// Ready Trader Go is free software: you can redistribute it and/or
	// modify it under the terms of the GNU Affero General Public License
	// as published by the Free Software Foundation, either version 3 of
	// the License, or (at your option) any later version.
	//
	// Ready Trader Go is distributed in the hope that it will be useful,