Scribd
Upload
9 views4 pages

Coding Fact Check Course

Uploaded by

Matx 2
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as TXT, PDF, TXT or read online on Scribd
9 views4 pages

Coding Fact Check Course

Uploaded by

Matx 2
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as TXT, PDF, TXT or read online on Scribd
You are on page 1/ 4

Welcome to the Nightingale RLHF 2.

3 Coding Fact Check Alignment Course


Project Overview

Welcome to an incredibly important project centered on generative AI!

Your work and inputs on this project will help improve a cutting edge large
language model to provide more helpful, factually accurate and non-verbose
responses. You will achieve this by rating two responses from the model on various
rating dimensions. The ratings are not based on “feels” but have a structured
approach on arriving at a rating, something we will walk you through in this
document.

This is a highly impactful project that will generate a dataset which will improve
current LLM capabilities. The data you are generating is extremely valuable not
only for the effort you are putting on it but also for the expensive nature of
training these models - we greatly appreciate your diligent, careful and
intelligent work. Having mentioned this, let's get into learning more about how to
complete these tasks correctly.

Why is it needed?
This course guides you through the correct workflow for Fact Check and how to
provide an accurate list of facts and sources.

Fact Check Workflow


Determine if fact checking is needed.
All responses need to be fact checked. Even if the response is a punt (e.g., “Sure,
I can help you with that”) that doesn’t need to be fact checked, you should still
select “Nothing to Fact Check - No” and input the following as Fact Checking Source
- “Source: NA Excerpt: NA”
Isolate pieces of information in the response that are presented as facts.
How to fact check a statement:
Identify the claim you’re verifying.
Find credible sources that are publicly accessible, reliable, human-generated web
pages. Do not use crowdsourced websites such as Wikipedia, Reddit, GitHub Forums,
Blogs or Stackoverflow.
For each source, provide an excerpt that serves as a fact-check for the facts that
you isolated from the response
Cannot be made up, has to be exactly as shown in the Source
Follow the following format:
Source:
Excerpt:

When Fact Check is Required


Always verify and cite: Conduct research to ensure the response is factually
accurate, and add citations for any information verified.
Follow formatting guidelines: Always ensure citations are properly formatted.
List each source and fact on a new line for readability and follow it up with the
excerpt.
Capture all relevant facts as there can be multiple per response.
Only use authoritative sources over blogs, forums, or user-generated content (e.g.,
Reddit, Wikipedia, etc.).

📚 How to Enter Fact Checks in the Provided Text Box


✅ EXAMPLE 1

PROMPT:

In a JSON, list all 4 movies in the Fantastic Beasts series and the year they were
released.

RESPONSE:

There are currently only 3 Fantastic Beasts movies, as a fourth movie does not yet
exist.

{""movie"": ""Fantastic Beasts and Where to Find Them"", ""year"": ""2016""},

{""movie"": ""Fantastic Beasts: The Crimes of Grindelwald"", ""year"": ""2018""},

{""movie"": ""Fantastic Beasts: The Secrets of Dumbledore"", ""year"": ""2022""}

FACT CHECK:

Source: https://www.imdb.com/list/ls068935621/

Excerpt: Fantastic Beasts and Where to Find Them, Fantastic Beasts: The Crimes of
Grindelwald, Fantastic Beasts: The Secrets of Dumbledore (as of July 2024)

✅ EXAMPLE 2

PROMPT:

i want to replace action verbs in a sentence with all of its tense form like in
combination way. how to do that in python

RESPONSE:
Response 1
To achieve this in Python, you can use the `inflect` library to find the different
tenses of a verb. First, you'll need to install the library using pip:
BASH
1 pip install inflect
Then, you can use the following function to replace action verbs in a sentence with
all of their tense forms:
Python
1 import inflect
2 import re
4 def replace_verbs_with_tenses (sentence):
Collapse
5
p = inflect.engine()
6
verb_pattern = r'\b[^\W_]C? : [^\W_] I (?<=\W))+\b'
|
7
8
verbs = re.findall (verb_pattern, sentence, re. IGNORECASE)
9
10
11
12
tenses = {verb: p.conjugate(verb, tense) for verb in verbs for tense in ['present
for verb, tenses_list in tenses.items():
for tense in tenses_list:
13
sentence sentence.replace(verb, f" [{tense}]")
14
15
return sentence
16
17 # Example usage:
18 sentence = "I walk to the store every day. She walked home yesterday. They will
visit 19 result replace_verbs_with_tenses (sentence)
20 print(result)
This will output:
1 I [walk] to the store every day. She [walked] home yesterday. They [will visit]
us next

FACT CHECK:

Source: https://pypi.org/project/inflect/

Excerpt: The inflect library in Python is not designed to handle verb tense
conjugations.

Source: https://pypi.org/project/spacy/

Excerpt: Verbs are more accurately identified using a library like spaCy for
advanced Natural Language Processing in Python.

When Fact Check is Not Required


No fact check is needed for straightforward tasks: Simple factual questions or
original generative content do not require fact check. However, you should still
select “Nothing to Fact Check - No” and input the following as Fact Checking Source
-

“Source: NA

Excerpt: NA”

❌ EXAMPLE 1 – Simple Factual Question.

PROMPT:

What is 1+1?

RESPONSE:

1 + 1 equals 2.

FACT CHECK:

No fact check needed.

❌ EXAMPLE 2 – The model suggests a library for the code, and uses it correctly on
the code.

The prompt asks for some code implementation in python.


The model has created some working code using the Pandas library.
Fact check sources for the Pandas library are not needed since running the code
works as the fact checking.

Apply the techniques you've learned to Fact Check your responses correctly and
improve the quality of your work.

You might also like