If you're changing the selected item from a button click, then you'll need to change the focus back to the listview for the item to be highlighted in blue:

listView1.Focus();

private void btnAdd_Click(object sender, EventArgs e)
{
    ListViewItem item = new ListViewItem(textBox1.Text, 0);
    item.SubItems.Add(textBox2.Text);
    item.SubItems.Add(textBox3.Text);
    item.SubItems.Add(textBox4.Text);
    listView1.Items.Add(item);
    item.Selected = true;
    decimal quantity;
    decimal.TryParse(item.SubItems[2].Text, out quantity);
    numericUpDown1.Value = quantity;
    listView1.Focus();   
}

private void btnDelete_Click(object sender, EventArgs e)
{
    if (listView1.SelectedIndices.Count > 0)
    {
        listView1.Items.RemoveAt(listView1.SelectedIndices[0]);
    }
}

private void listView1_SelectedIndexChanged(object sender, EventArgs e)
{
    if (listView1.SelectedIndices.Count > 0)
    {
        decimal quantity;
        decimal.TryParse(listView1.SelectedItems[0].SubItems[2].Text, out quantity);
        numericUpDown1.Value = quantity;
    }
}

private void numericUpDown1_ValueChanged(object sender, EventArgs e)
{
    if (listView1.SelectedIndices.Count > 0)
    {
        ListViewItem item = listView1.SelectedItems[0];
        item.SubItems[2].Text = numericUpDown1.Value.ToString();
        decimal pricePerUnit;
        decimal.TryParse(item.SubItems[1].Text, out pricePerUnit);
        item.SubItems[3].Text = (pricePerUnit * numericUpDown1.Value).ToString();
        listView1.Focus();
    }
}

private void btnDelete_Click(object sender, EventArgs e)
{
 
    if (listView1.SelectedIndices.Count > 0)
    {
        listView1.Items.RemoveAt(listView1.SelectedIndices[0]);
    }
    if (listView1.Items.Count > 0)
    {
        listView1.Items[listView1.Items.Count -1].Selected = true;
        listView1.Focus();
    }
    else
    {
        numericUpDown1.Value = 0;
    }
}

private void btnDeleteAll_Click(object sender, EventArgs e)
{
   listView1.Items.Clear();
   numericUpDown1.Value = 0;
}

To my knowledge, it's not possible to embed other controls in a ListBox and so, if you want to do this, you'd need to consider using a different kind of control to display your items.

You might be able to embed a NumericUpDown control in a ListView (see for example this article) but it won't be easy.

However, the best solution for embedding other controls and dealing with columnar editing is the DataGridView and there's an article here from MS themselves showing how to build a custom cell and column based on a NumericUpDown.

Would it be easy to do the embedding of NumericUpDown control inside ListBox, if I switch to WPF, instead of WinForm? or is it just as difficult?

I do not know why I could not add comment. yes, ListBox does have its limitations, I am converting to ListView, which is easier and more flexible.

anyway, for ListBox, I want to hightlight the latest item added, I have this:

listBox1.SelectedIndex = listBox1.Items.Count - 1; // to always hightlight blue latest item added.

for ListView, how do I achieve the same to always highlight blue the latest item added?
I try this, but does not work.
listView1.Items[0].Selected = true;

yes, I understand that having a single standalone NumericUpDown, adjacent to the ListView is simpler to implement.

However, for my application, that number of rows in the ListView is a variable. When user clicks some button, it adds on to the row. User could also delete the whole row from the ListView.

So, I think the best suitable method would still embed NumericUpDown control inside ListView, which is not easy.

hi Vulpes, your code works wonderfully. thank so much for your kind help. My account, I could not add comment nor Set as Accepted Answer, so I can only add answer to THANK YOU.